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ABSTRACT 

Two student evaluation surveys developed in the 
Onited States were administered to a sample of University of Sydney 
students to determine their applicability in Australia. The Endeavor 
Instructional Rating Form (Frey et al.l measures seven components of 
effective teaching that have been demonstrated with the use of factor 
analysis i-n several different settings^ and ratings on this form have 
correlated with student learning. The SSSQ survey (Students' 
Evaluation of Educational Ou-ality) by Marsh has been factor analyzed 
and nine factors have -been identified (learning/value, enthusiasm, 
organization/clarity^ group interaction, rapport, breadth of 
coverage, examinations/grading, assianments, workload/diff iculty) . 
Students were asked to evaluate one of the best and one of the worst 
lecturers in th-^ir university experience, to indicate inappropriate 
items, and to select the most important items. Each of the 63 items 
was seen as appropriate by most students, each item was chosen by at 
least a few students as being most important # and all items, except 
those related to workload/difficulty, differentiated between best and 
worst lecturers. Separate factor analyses of the two surveys repealed 
the same factors that had been identified in' American settings. 
Furthermore, there was good agreement between the factors from the 
two surveys that were hypothesized to measure the same components of 
effective teachina. It is' concluded that the findings demonstrate the 
feasibility of evaluating effective teaching in Australian 
universities and the appropriati^ness of two American surveys to an 
Australian setting* A list of the factors in each survey, the results 
of oblique factor analysis, and reference-^ are included. 
(Atithor/SV?) 
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student Evaluations 



Jft)stract 

Two student evaluation surveys developed in the Unite<5 , states were . administered 
to a sample of Australian university students. Stude^its were asked to evaluate 
one of the best and one of the worst lecturers in their ijniversity o.xperience , 
to indicate inappropriate items, and to jselect the) most important items. Each 
of the 63 items was seen as appropriate by most students, each item was chosen 
by at least a few students as beisig most important, and all items — except those 
related to Workload/Difficulty — differentiated between best and worst lecturers 
Separate factor analyses of the two surveys revealed the same factors that had 
been identified in American settings. Furtliermore, there was good agreement 
between the factors from the two surveys that were hypothesiized to K^^asure the 
same components of effective teaching. The findings demonstr^^te the feasibility 
of evaluating effective teaching in Australian universities and the appropriate- 
ness of two American surveys to an Australian setting. 
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students' Evaluations 2 
Students' Evaluations of Tertiary Instruction: 
Testing the Applicability of American 
Surveys in an Australian Setting 

* Students' evaluations of teaching effectiveness are commonly collected 
at American institutions of higher education, and are widely endorsed by both . 
students and faculty {Centra, 1979; Astin & Lee, 1966; Seldin, 1975). The 
purposes of these evaluations are variously to provide: 1) a source of diagnostic 
feedback to faculty about the effectiveness of their teaching; 2) a measure of 
teaching effectiveness to be used in tenure/promotion decisions; and, 3) a source 
of information for students to use in the selection of courses and lecturers. 
While the first purpose is nearly universal, the second two are not. At some 
universities student input is required before faculty are even considered for 
promotion, while at others the inclusion of students' evaluations in such 
deliberations is optional. Similarly, the results of student ratings are 
published and sold in campus bookstores at some universities, while at others the 
results are considered to be strictly confidential. 

Ttie use of students' evaluations, especially for tSniife/promotion decisions, 
has not been without opposition, and investigation of different aspects of the 
ratings has stimulated considerable research. Particularly in the last few years 
this has become one of the most frequently studied areas in American "educational 
research* In contrast to the wide use of student ratings in the United States 
and Canada, they apparently have not been systematically collected in either 
the United Kingdom or Australia (Smith, 1980). Fiarthermore, there has been 
no attempt to empirica] ly test student ratings or the applicability of rating 
instruments developed in the United states. The piarpose of this study is to 
describe two such instruments, and to report upon an investigation of their 
applicability in an Australian setting. 
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Ihe two evaluation instruments used in this study are the SEEQ survey 
(Students' Evaluations of Educational Quality) developed by the autlior 
(see Marsh, 1980b; in press) and the Endeavor Instructional Rating Form devised 
by Frey (Frey, 1978; Frey, Leonard & Beatty, 1975) . While other instrments have 
been developed (see Centra, 198Q |or a summary) , both these multif actor 
instruments were • explicitly developed to measure separate components of 
effective teaching, and both have^been shown to be reliable, valid, and 
little affected by -a variety of sources of potential bias. 

Ihe Endeavor form measures seven components of effective teaching that 
have been demonstrated with the use of factor analysis in several different 
settings • (Frey , Leonard & Beatty, 1975). Frey has shown that ratings on 
Endeavor are correlated with student learning (Frey, 1973? 1978; Frey, Leonard 
& Beatty, 1975). In these studies, as well as similar studies conducted with 
SEEQ, student ratings are collected in large multisection courses. Different 
sections of the same course are taught by different instructors, but each 
section is taught according to a similar course outline, has similar goals and 
objectives, and most importantly is tested with the same standardized final 
examination. Ihose sections that rate teaching to be most effective near the 
end of term are also the sections that perform best on the final exeimination. 
Frey (1978) has also argued that "pedagogical skill", as measured with the 
Endeavor form, is relatively unaffected by class size, expected course grade, 

and instructor rank, but is correlated with research productivity • However, 

I 

ratings of "rapport" (class discussion and availability for individual attention) 
were more strongly affected by class size and e^ipected graders, but were not 
related to research productivity. 

SEEQ and the research that led to its development has been recently 
summarized (Marsh, 1980b) . Factor analysis has identifed the nine SEEQ factors 
in several different studies (Marsh, 1978; in press) . Furthermore, factor 
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students' Evaluation 4 

analysis of faculty self evaluations of their own teaching effectiveness with the 
SEEQ form also indentified these same factors (Marsh, in press; also see Marsh, 
Overall & Kesler, 1979) • Various indicies of reliability, including intraclass 
correlations and coefficient alphas, all indicate that the reliability of these 
factors is generally at least ,9 (Marsh, 1980b), Additionally, when the same 
students were asked to reevaluate teaching effectiveness several years after 
graduation from their \iniversity programs, the average correlations — based on 
ratings of 100 classes — was ,83 (Overall & Marsh, 1979a). Ratings on SEEQ have 
been successfully validated against the ratings of former students (Marsh, 1977), 
student learning as measured by objective examination (Marsh, Fleiner & Thomas, 
1975; Marsh & Overall, 1980), affective course consequences such as the application 
of the course materials and plans to pursue the subject further (Marsh & Overall, 
1980), and faculty self evaluations of their own teaching effectiveness (Marsh, 
Overall & Kesler, 1979; Marsh, in press). None of a set of 16 "potential biases" 
(e.g., class size, expected grade, and prior subject interest) could account for 
more than 5% of the variance in SEEQ ratings (Marsh, 1980a), and many of the re- 
lationships were inconsistent with a simple bias explanation (e.g., more difficult 
courses that require more time outside of class were rated more favorably) . 

Finally, feedback from SEEQ, particularly when coupled with a canSid discussion 
with an external consultant, led to improved ratings and better student learning 
(Overall & Marsh, 1979). 

In the present investigation items from both SEEQ and the Endeavor form 
were administered to a broad sample of Australian tertiary/miversity students. 
Students were asked to evaluate the teaching effectiveness of a best and a worse 
lecturer, to indicate inappropriate items, and to select the most important 
items. These criteria, in addition to factor analysis of the ratings, were used 
to test the applicability of these American instruments in an Australian 
setting. 

ERIC ^ 
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Methodology 

Sample 

The evaluation survey was administered to a total of 158 students 
currently enrolled at the University of Sydney, Approximately 30% of these 
students were enrolled in a course on Human' Growth and Development, an 
introductcry level co\irse offered in the Faculty of Arts. The remainder were, 
recruited on an ad hoc basis in various campus libraries, the student union, and 
departmental lounges. Students were asked to participate, given a brief 
description of the study, guaranteed of the confidentiality of their responses, 
and ensured that they wotold not be asked to identify themselves in any way.. 
Students, if they agreed to participate, wez/e given a self-addressed, stamped 
envelope with the survey, and asked to return it within three weeks. Approximately 
half of the surveys actually were returned, and represented a broad cross section 
of university students. Students were asked to indicate the department in which 
•they would receive their degree/ and a total of 25 different departments were 
named. Ihe courses that were actually evaluated came from 30 different 
departments . 

S iarvey ' 

The evaluation survey consisted of three pages. The first page consisted 
of instructions and demographic items, and requested that students select one 
of the best and one of the worst lecturers that they had experienced at the 
University of Sydney. Students were asked to limit their choices to "lecturers 
v*io were primarily 7-esponsible for an instructional sequence that lasted at 
least one term". Students were then instructed to fill out two separate 
questionnaires^ one for the best lecturer and one for the worst - each containing 
63 evaluation items. Items were to be answered on a fiye-point response scale 
that varied between "1-Very Poor or Very Low or Almost Never" and "5-Very Good 
or Very High or Almost Always". An additional "not appropriate" response 
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categorey was provided for items that were not relevant to the course being 
evaluated (items left blank were also counted as "not appropriate" responses) , 
After completing the ratings for a given lecturer, students were asked to select 
up to five questions that they "felt were most important in describing either 
positive or negative aspects of your overall learning experience in this 
instructional sequence." 

Statistical Analysis 

Each of the evalu5:tion items was initially tested in terms of: 1) its • 
ability to discriminate between best and worst lecturers; 2) its appropriateness 
ti-ec,. the lack of "not appripriate" responses); 3^ its importance (i.e., the 
number of "most important" nominations); and, 4) how well it correlated with other 
items designed to measiare the same component of effective teaching. Separate 
factor analyses were performed on the SEEQ and Endeavor forms. Factor scores 
derived from these factor analyses were correlated to determine the relationship 
between SEEQ and Endeavor factors. Finally a set of best items best in terms 
of factor analysis, discrimination, appropriateness, and importance — was 
selected from the entire set of items - 

. ^ Results and Discussion 

Evaluation of Individu al Items 

Preliminary inspection of the SEEQ and Endeavor surveys revealed considerable 
overlap in the factors defined by each. Five SEEQ factors (Learning/Value, 
Group Interaction, Individual Rapport, Examinations, and Workload/Difficulty) 
correspond closely to five Endeavor factors (Student Accomplishments, Class 
Discussion, Personal Attention, Grading, and Workload). A sixth SEEQ factor, 
Organisation/Clarity, has been divided into two factors on the Endeavor form 
(Presentation Clarity and Organisation/Planning). Three SEEQ factors. Instructor 
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Enthusiasm, Breadth of Coverage, and Assignments/Readings, do not correspond to 

any of the Endeavor factors. On the basis of this preliminary inspection the 

set of 34 SEEQ items and 21 Endeavor items were divided' into 10 content areas • 

that were hypothesized to correspond to 10 components of effective teaching (see 

Table 1) . In addition to these 55 items , eight additional items were 

also classified into one of the 10 categories to make a total of 63 items. 

Students evaluated a best and a worst lecturer with the entire set of 

63 items. The best lecturers were evaluated more favorably (p{,00l) on al,l items 

except those in the Workload/Difficulty factor (See Table 1) . The differtsnces 

were largest for the LeamingAalue/Accomplishmcnt, Instructor Enthusiasm, and 

Presentation Clarity factors. While there was little difference between best 

and worst lectureirs in terms of the difficulty of courses which they teach, 

courses taught by the best lecturer were judged as more difficult on five or six 

items (one to a statistically significant e::tent) . These findings demonstrate 

that both the SEEQ and the Endeavor items are able to clearly discriminate 

between lecturers that Australian students have selected as being best and worst. 

Insert Table 1 About Here 

Students were specifically asked to indicate items that were inappropriate 
for evaluating one of their lecturers or the course he/she taught. Only two of 
the 63 items were judged to be inappropriate (including those that left an item 
blank) by more than 10% of the students, (see Table 1) One of these concerned 
the availability of personal atte^ntion (some Australian students did not seek it) , 
and the other asked about feedback from eixaminations (some courses did not 
administer exams or administered only a final examination tlmt was not returned 
to students). In general, items falling into the Group Interact.i.on/Discussion , 
Individual Rapport/Personal Attention, and Gradinq/Examination factors were most 
frequently seen as inappropriate. However, every single item was judged to be 
appropriate by at least 80% of the students, and most were appropriate for 95% 
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or more of them. These results indicate that the SEEQ and Endeavor items are 
■ appropriate in an Australian setting. 

Students selected as many as five items that they felt were most important 
in describing either positive or negative aspects of the overall learning 
experience. Each of fJie 63 items (see Table 1) received at least three "most 
important" nominations, and at least one item in each of the 10 categories 
received 17 or more nominations. The three most frequently selected items) were 
"teaching stylo held your interest-'" -"presented clearly and summarized", ari'd>~ 

"was enthusiastic about teaching". Items falling into the Instructor Enthusiasm 

and Presentation Clarity factors were nominated most frequently. While some of 
the items and some of the factors appear to be more important, t:he nominations 
were spread widely over the entire set of i terns • This suggests tihat each item 
measures a potentially important component of effective teaching. 

For each item, the average correlation between that item and other items 
in the same factior, and between 1:hat item and all otiher items was computed 
(see Table 1) . Items that do not correlate witih any otiher items are probably 
not related to effective teaching in an Australian setting. Items tihat correlate 
as highly witih all items as wilih items designed to measure the same component 
of effective teaching lack specificity, though they may measure some generalized 
notion of effective teaching. Inspection of Table 1 indicates that every item' 
is substantially correlated wilih oliher items lihat measure 1:he same factor. 
However, most of 1:he items also have substantial correlations wilih tihe entire 
set of items, and a few are more highly correlated with the entire set of items 
than with items specifically designed to measure the same component. This 
suggests t:hat there is a large general factor underlying the student ratings, 
and that the different factors are highly correlated. While this large general 
factor or halo effect is undesirable, the manner in which the data was collected 
may have produced this occurence. Students were specifically asked to select 
their "best" and "worst" lecturers, suggesting that most lecturers would be 

in 
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judged as "generally good" or "generally bad". A smaller general factor might 
be expected if students had also considered lecturers who fell between the two 
extremes. Nevertheless, the fact that most items were more highly correlated 
v/ith other items within the 'same factor suggests that the separate factors 
are meaningful. Ohe test of this tentative appraisal will be the results of the 
factor analyses described below. 

Factor Analysis of the SEEQ and Endeavor Surveys 

The items from Marsh's SEEQ survey (those marked "M" on Table 1) were 
factor ^analyzed (see Table 2). Marsh (1978; in press) has demonstrated that 
the survey measures nine separate components of effective teaching in American 
settings. Factor analysis of the ratings by Australian students clearly 
identify eight of these factors. Except for two items from the Examinations/ 
Grading factor, each item loaded substantially on the factor it was designed to 
measure and had smaller loadings on each of the other factors. Only one of the 
Examination items loaded substantially on that factor, aiiese results show that 
at least eight of the nine SEEQ factors are well defined in an Australian 
setting. 

Insert Table 2 About Here 

A separate factor analysis of the 21 items from Frey's Endeavor 'survey 
(those marked "F" in Table 1) was also performed. The results of this analysis 
clearly identifies each of the seven Endeavor factors (see Table 3) . With one 
exception (item F13) , each item loads substantially on the factor it was designed 
to measure and has smaller loading on each of the other factors. Frey's Grading 
factor, unlike the Examination factor from SEEQ, appears to be well defined. The 
Endeavor factor emphasizes the determination f grades rather than the actual 
examinations. The Endeavor factors that are most highly correlated. Clarity and 
Organization/Planning, are the two factors that are combined into' a single factor 
on the SEEQ iorm. While these factors are highly correlated, the correlation is 
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loss than the reliability of either (coefficeint alpha ~ see Table 4) , thus 
supporting their separation. Th$se findings demonstrate that all seven of the 
Endeavor factors axe well defined in an Australian setting. 



Insert Table 3 About Here 

Factor scores, weighted averages of the standardized ratings for each item 
(Nie, et al. , 1975), were corr»puted for the nine SEEQ factors and the seven 

s 

Endeavor factors. Correlations among these different factor scores are presented 
in Table 4. This set of correlations is somewhat analagous to a multitrait- 
mialtimethod matrix in which the different factors correspond to the multiple 
traits and the different surveys are the multiple methods. Criterion developed 
by Campbell & Fiske a959; also see Marsh & Hocevar, 1980) can be applied to test 
for convergent and discriminant validity. Convergent validity refers to the 
extent oJ agreement between the two different surveys.. 

Discriminant validity refers to the distinctiveness of the 'different 
factors; specifically, do the highest correlations appear between the SEEQ and 
Endeavor factors that were designed to measure the same component of effective 
teaching. 

Insert Table, 4 About Here 

The application of the multitrait-multimethod analysis is somewhat 
complicated by the fact that the SEEQ and Endeavor suiiveys were not specifically 
designed to measure the same components of effective teaching. 

However, the four criteria proposed by Campbell and Fiske can be adapted. 

i 

They are: 

1) Agreement on SEEQ and Endeavor factors that match should be substantial 
(a criterion of convergent validity) 

2) Correlations between these matching factors should be higher than the 

12 
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correlations among different SEEQ factors or among different Endeavor 
factors (a criterion of discriminant val 
3) Correlations between these matching factors shoiald be higher than 

correlations between other SEEQ and Endeavor factors that were not designed 
to measure the same component (a second criterion of discriminant validity) • 
Ttie pattern of correlations among SEEQ factors should be similar to the 
pattern among Endeavor factors (e.g. if the two SEEQ factors Group 
Interaction and Individual Rapport are highly correlated, then so should 
the corresponding Endeavor factors) . Satisfaction of this criterion 
V suggests that correlations amon.g different factors represent correlations 

between the underlying traits being assessed rather than a method or halo 
effect.. 

On the basis of the preliminai^fT inspection of the two surveys, each of the 
seven Endeavor factors was predicted to correspond most closely to one of the 
SEEQ factors (the Organization/Clarity factor on SEEQ was hypothesized to be a 
combination of the Organization/Planning and Clarity factors on Endeavor) . Each 
of these seven correlations is substantial and statistically significant 
(median r = .80), th\as supporting their convergent validity. Six of these 
correlations — the one exception being the correlation between the Grading 
(SEEQ) and Examinations (Endeavor) factors — are higher than other 
correlations in the mialtitrait-raultimethod matrix, supporting the 

discriminant validity of the factors. 

In spite of the good evidence for both convergent and discriminant 
validity, correlations among the different factors are larger than desirable. 
Among the SHEQ factors, correlations between Learning/Value, Instructor Enthusiasm, 
and Organization/Clarity factors are all over .50, as is the correlation between 
Group Interaction and Individual Rapport. Among the Endeavor factors Organization/ 
Planning and Clarity are correlated .67, while correlations between Personal 
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Attention and Group Discussion, Organization/Planning and Student Accomplishments, 
and Clarity and Student Accomplishments are all about .6. Several points are 
relevant to interpreting these high correlations. First, as previously mentioned, 
part of these high correlations might be a function of the manner in which the data 
was collected. Secondly, each of these correlations is substantially less than 
the reliabilities of the factors. Third, there was a similarity in the pattern 
of correlations among SEEQ factors and among Endeavor factors (Campbell & Fiske's 
fourtli criterion — also see Marsh & Hocevar, 1980) . This final point suggests 
that the underlying dimensions being measured may in fact be correlated in a way 
that is iiidependent of the method of data collection. Nevertheless, there is 
still the possibility of a substantial method or halo effect that should be 
explored in future research. 

In summary, SEEQ and Endeavor factors that have been demonstrated in 
American settings were also clearly identifed iji the evaluations by Australian 
students. Th^ mioltitifait-meltimethod analysis of the two surveys offered good 
evidence for both their convergent and discriminant validity. This argues for 
the miolti dimensionality of student ratings, the distinctiveness of the different 
factors measured by SEEQ and Endeavor, and the generality of these factors beyond 
the specific items included on each instrument. 

A Combination of the Two Surveys 

A set of 43 items was selected to best define the 10 hypothesized factors. 
Items were selected primarily on the basis of factor analysis, but criteria such 
as discrimination between best and worst lecturers, appropriateness, and most 
importaiit nominations were also considered. Factor analysis (see Table 5) provides 
good support for the 10 -factors that were hypothesized. Every item loads a1. least 
.35 on the factor it was hypothesized to measure (most were over»5), and no item 
loaded higher than .35 on any other factor (most were under. 20). The clarity 
of this factor analysis offers further support for mialtidimensionality of 
student ratings, and the generality of the evaluation factors as measured by the 
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SEEQ and Endeavor instruments • 

Insert Table 5 About Here 

Signmary and Implications 

Items from two student evaluation surveys developed in the United States 
were administered to a sample of Australian university students. These students 
were asked to sel^t a best and worst lecturer from their university experience, to 
evaluate both with the combined survey, and to select the items they felt were 
most important. Each of the 63 items was seen as appropriate by at least 80% 
of the students (most by 95% or more) , each item was selected by at least a few 
students as being most important, and all the items — except those in the Workload/ 
Difficulty factor — differentiated well between the best and worst lecturers. 
Separate factor analyses of the items from the two surveys identified the factors 
each had been designed to measure. Furthermore, there- was good agreement between 
factors from the > two different surveys th^t were hypothesized to measure the same 
components of effective teaching. Finally, a factor ''analysis of items from both 
surveys offered even clearer support for the hypothesized set of evaluation factors. 

These findings clearly demonstrate that teaching effectiveness can be 
measured in an Australian setting, that ev£j.uation forms developed in the United 
States are appropriate to an Australian setting, and that the same components that 
underlie effective teaching in the United States are also relevant in Australia. 
The findings cilso suggest that the extensive range of research dLnto student 
ratings that has been conducted in the United States may also apply to Australian 
settings. Future research, perhaps employing the surveys used in this study, is 
needed to determine the extent of the validity of student evaluations of teaching 
effectiveness in Australian settings, and to document problems involved in the 
actual implementation of such programs. 
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Hypothesized Factors, Individual Items and Their Characteristics (N=s 316 "best" and "worst" lecturers) 



Evaluation Items (paraphrased) 
and Hypothesized Factors 

LEARNING/V^UE/ACCOMPI>ISHMENTS 

Ml Course challenging & stimulating 

M2 Learned something valuable 

M3 Increased subject interest 

M4 Learned & understood subject matter 

M30^ OVERALL COURSE RATING 

F19 Understood .advanced material 

F20 Ability to analyze issues 

F21 Increased knowledge & competence 

03 Interest after completion of course 

INSTRUCTOR ENTHUSIASM 

M5 Enthusiastic about teaching 

M6 Dynamic and energetic 

M7 . Enhanced presentation with humor 

M8 ^ Teaching style held your interest 

M31 OVERAIIi INSTRUCTOR RATING 

PRESENTATION CLARITY (Organization)^ 

M9 Lecturer explanations clear 

MIO Materials well explained & prepared 

M12 Lectures facilitated taking notes 

Fl Presentations clarified materials 

F2 Presented clearly & summarized 

F3 Made good use of examples 

PLANNING/OBJECTIVES (Organization)^ 

Mil Course objectives stated & pursued 

F13 Presentations planned in advance 

F14 Provided detailed course schedule 

F15 Activities orderly scheduled 

01 Time distibuted over topics appr-'€ly 

07 Aims & objectives clear to students 

08 Announced goals s/or criteria 

GROUP INTERACTION/DISCUSSION 

M13 Encouraged class discussions 

M14 Students shared knowledge/ideas 

M15 Encouraged questions & gave answers . 

M16 Encouraged expression of ideas 

Flo Class discussion was welcome 

Fll Students encouraged to participate 

F12 Encouraged students to express ideas 

INDIVIDUAL RAPPORT/PERSONAL ATTENTION 
M17 Friendly towards individual students 
M18 Welcomed students seeking help/advice 
M19 Interested in individual students 
M20 Accessible to individual students 
F7 • Listened & was willing to help 
F8 Able to get personal attention - 
F9 Concerned about student difficulties 



Discrimination 



Mean of 
"Best" 



Mean of 
"Worst 



t-value 



BREADTH OF COVERAGE 
M21 Contrasted various implications 
M22 Gave background of ideas/concepts 
M23 Gave different points of view 
M24 Discussed current developments 

GRADING/EXAMINAT IONS 

M25 Examination feedback valuable 

M26 Evaluation methods fair /appropriate 

M27 Tested course content as emphasized 

F16 Grading was fair & impartial 

F17 Grading reflected student performance 

F18 Grading indicative of accomplishments 

06 Exams covered materials broadly 

ASSIGNMENTS/READINGS 

M28 Readings/texts were valuable 

M29 They contributed" to understanding 

02 They encouraged further exploration 

C4 Assignment /reading expectations clear 

05 They were integrated ii to the course 

WORKI/DAD/DIFFICULTY 
•M32 Course "difficulty (Easy-Vard) 
Course workload (Light-Heovy) 
Course pace (Too Slow-Too Bast) 
Students had to work hard 
Course required a lot of time 
Course workload was heavy 



M33 
M34 
F4 
F5 
^F6 



Number of 
"Not Appro- 
priate" 
responses 



Number of 
"Most Im- 
portant" 
nominations 



Correlation 
with other 

items on 
same scale 



Correlation 
with all 
other 
items 



4.16 


2.32 


17.26*** 


1 


63 


.764 


.728 




Z . Jo 


15 . 51*** 


2 


51 


. 823 


.790 


3.92 


1.94 


17.53*** 


4 


30 


.846 


.807 


4.08 


2.34 


17.62*** 


1 


16 


.821 


.792 


4.35 


2.20 


21.32*** 


0 


32 


.852 


.828 


3.98 


2.48 


12.45*** 


11 


16 


.664 


.775 


3.96 


2.15 


17.38*** 


3 


32 


.856 


.802 


4.19 


2.36 


18.97*** 


2 


26 


.856 


.783 


4.17 


2.10 


20.19*** 


3 


24 


.851 


.813 


4.54 


2.41 


19-76*** 


1 


88 


.807 


.784 


4.25 


1.75 


25-20*** 


1 


63 


.900 


.839 


3.99 


1.74 


18.76*** 


4 


50 


.793 


.771 


4.49 


1.25 


45.64*** 


1 


116 


. 898 


.913 


4.67 


1.55 


45.15*** 


1 


39 


.887 


.884 


4.41 


1.70 


32-22*** 


2 


67 


.902 


.811 


4.22 


1.88 


25.10*** 


4 


34 


.874 


.852 


4.38 


1.78 


25.67*** 


2 


53 


.815 


.778 


4.29 . 


1.58 


32.68*** 


0 


62 


.919 


.841 


4.54 


1.77 


33.41*** 


0 


90 


.927 


.847 


4.24 


2.32 


18.51*** 


2 


22 


.785 


.767 


4.15 


2.15 


19.55*** 


7 


30 


.789 


.804 


4.45 


3.11 


11.99*** 


2 


42 


.672 


.695 


3.83 


2.58 


9.14*** 


3 


18 


.627 


.491 


3.90 


2.43 


13.28*** 


28. 


4 


.756 


.760 


4.14 


2.89 


11.56*** 


6 


9 


.645 , 


.671 


4.06 


2.10 


18.57*** 


3 


21 


.805 


.772 


3.79 


2.29 


13.54*** 


5 


9 


.789 


.733 


3.50 


2.49 


9.91*** 


32(10%) 


35 


.903 


.755 


3. 33 


1.99 


11.21*** 


25 


12 


.856 


.763 


3.79 


1.96 


17.39*** 


7 


12 


.796 


.836 


3.44 • 


2.14 


11.71*** 


12 


10 


.887 


.792 


3.65 


2.34 


10.16*** 


31 


6 


.872 


.735 


3.36 


2.20 


9.02*** 


32(10%) 


35 


.841 


.879 


3.37 


2.10 


11.02*** 


18 


3 


.919 


.782 


3.91 


2.51 


12.18*** 


10 


23 


.820 


.713> 


3.86 


2.37 , 


: 13.34*** 


7 


24 


.884 


.788 


3.76 


2.03 


15.37*** 


15 


10 


.857 


.775 


3.56 


2.45 


10.02*** 


21 


10 


.789 


.717 


3.97 


2.43 


13.70*** 


21 


16 


.836 


.759- ' 


3.94 


2.70 


10.74*** 


58(18%) 


18 


.786 


.745 


3.62 


2.14 


13.40*** 


9 


21 


. 849 


• / 


4.00 


2.54 


12.68*** 


18 


11 


.703 


.672 


3.74 


2.47 


11.91*** 


10 


14 


.746 


.713 


3.87 


2.65 


11.09***. 


27 


22 


-716, . 


.727 


3.81 


2.29 


13.37*** 


17 


9 


.709 '* 


.785 : 


3.49 


2.21 


10.64*** 


41(13%) 


8 


.551 


.612 


3.77 


2.93 


8.06*** 


22 


17 


.700 


.593 


4.02 


2.97 


9.44*** 


26 


13 


.626 


.612 


3.97 


2.43 


13.70*** 


21 


16 


.785 


.654 


3.60 


2.76 


7.94*** 


31 


10 


.745 


.622 


3.60 


2.68 


9.36*** 


28 


6 


.765 


.659 


3.93 


2.89 


10.01*** 


23 


5 


.701 


.704 


3.48 


2-99 


3.81*** 


12 


10 


.484 


.399 


3.75 


2.87 


7.39*** 


2 


20 


.662 


.604 


3.41 


2.08 


11.24*** 


8 


12 


.616 


.711 


3.74 


2.78 


8.?9*** 


10 


4 


.597 


.695 


3.89 


2.89 


9.11*** 


14 


11 


.670 


.700 


3.17 


3.38 


- 1.90 


2 


8 


.693 


-.002 


3.23 


3.22 


0.11 


0 


20 


.718 


.'100 


3.14 


3.11 


0.21 


0 


23 


.512 


.020 


3.46 


3.41 


0.71 


0 


12 


.739 


.114 


3.32 


3.16 


1.31 


0 


5 


.800 


.159 


3.33 


3.03 


2.42* 


0 


3 


.811 


.300 



"rail rating items were not specif icf».lly designed to measure any one factor but previous researc 
shown that these are the factors that they are most related to in American studifss. 



ERIC 

^^"These two factors were combined into one ^on the Meursh survey but separated on the Frey survey. "I Q 



h (Marsh, 1980, in press) 



Table 2 

'Factor Analysis of SEEQ items <N» 316 sets of ratings) 

Factor Pattern Matrix 





Evaluation items (paraphrased) 


I 


TT 


TTT 
X^X 


-*V 


y 


VT 


VII 


VIII 


IX 


I 


LEARNING/ViiLUE 




















MI 


p3urse chadlenging s stimulating 


T57 


.20 


.04 


.03 


-02 


.05 


.19 


.06 


.11 


M2 


Learned something valuable 


.63 


.06 


.05 


.14 


-05 


.02 


.18 


.14 


.01 


W3 


Increased siibject interest 


,58 


.12 


.15 


.04 


.04 


.18 


-.01 


.09 


.02 


M4- 


Learned/Understood subject aiatter 


.40 


.07 


.34 


.11 


-.04 


.10 


.00 


.17 


-.14 


M30 


OVERALL COURSE RATING 


:49 


.15 


.15 


-.02 


.09 


.13 


.16 


.10 


-.04 


II 


ENTHUSIASM • 




















M5 


Enthusiastic about teaching 


.06 


.55 


. 10 


.11 


.09 


.11 


.10 


-.03 


-.05 


M6 


Dynamic & Energetic 


.04 


.69 


.08 


.07 


.06 


.08 


. *9 


. 01 




M7 


Enhanced presentations with humor 


.13 


.46 


.02 


.15 


.12 


.07 


.17 


.01 


-.11 


M8 


Teaching style held your interest 


.16 


.44 


.35 


.03 


.10 


.10 


.13 


-.05 


-.01 


M31 


OVERALL INSTRUCTOR RATING 


.23 


.40 


.30 


.06 


.11 


<13 


.10 


.01 


-.03 



III ORGANIZATION/CLARITY 



VQ 


Instructor Explanations Clear 


.07 


.25 


.56 


.03 


.13 


.02 


.17 


- 


,02 


- 


,08 


MIO 


course materials prepared & clear 


.11 


.20 


.57 


.02 


.08 


.15" 


.04 




.06 




.00 


Mil 


Objectives stated s pursued 


.28 


.06 


.36 


.00 


.13 


.15 


.20 




.06 




.03 


M12 


Lectures facilitated note taking 


.07 


.36 


.44 


.01 


.06 


.05 


.10 




.00 




.06 


IV 


GROUP INTEKACTION 
























M13 


Encouraged class discussions 


.04 


.07 


.00 


.79 


.08 


.05 


.02 




.10 




.01 


M14 


Students shared ideas/knowledge 


.02 


.08 


.10 


.70 


.08 


.06 


.07 




.06 




.01 


M15 


Encouraged questions & £mswers 


.11 


.05 


.27 


.36 


.31 


.11 


.10 


_ 


.10 




.06 


M16 


Encouraged expression of ideas 


.11 


.07 


.03 


.66 


.23 


.11 


.07 


- 


.08 




.01 


V 


INDIVIDUAL RAPPORT 
























M17 


Friendly towards students 


.15 


.12 - 


.12 


.10 


.67 


.20 


.06 


- 


.09 




.03 


:1L8 


Welcomed seeking help/advice 


.12 


.13 - 


.04 


.24 


.55 


.13 


.08 


- 


.02 


- 


.01 


M19 


Interested in individual students 


.11 


.16 


-08 


.12 


.58 


.21 




.01 


- 


.05 




.01 


M20 


Accessible to individual students 


-.08 


.14 - 


.01 


.18 


.49 


.17 


.03 




.18 




.09 


VI 


BREADm OF COVERAGE 
























M21 


Contrasted implications 


-.04 


.07 


.09 


.02 


.03 


.42 


.36 




.15 




.06 


M22 


Gave background of ideas /concepts 


-.01 


.08 


.10 


.22 




.13 


.50 


.20 




.12 




.07 


M23 


Gave different points of view 


-.06-. 01 


.14 


.07 


.08 


.64 


.11 




.13 




.01 


M24 


Discussed current developments 


.14 


.'13 


• 08 


.19 




02 


.38 


.15 




.14 




.01 


VII 
M25 


EXAMINATIONS /GRADING 
Examination feedback valuable 


.00-. 01 


.16 


.16 


.25- 


-.30 ■ 


TsT 




.30 




.09 


M26 


Eval. methods fair/appropriate 


.04 


.07 


.14 


.11 


.15 


.04 


.00 




.40 




.05 


M27 


Tested exphasized course content 


.08- 


.03 


.31 




.05 


.19 


.12 


.08 




.24 




.02 


VIII ASSIGNMENTS 
























M28 Reading/ texts vaJLuable 


.04-. 06 


.04 


.01 


.00 


.05 


.15 




.50 




.01 


M29 


Added to course understanding 


.15 


.21 - 


-.06 


.00 


.07 


.04 




.11 




.66 




.15 


IX 


WORKLOAD/DIFFICULTY 
























M32 


Course difficulty (easy-hard) 


-.15-. 01 


.03 


.09 




.03- 


-.02 




.03 




.05 




.84 


M33 


Course workload (light-heavy) 


.14- 


.03 


.06 




.04 


.06 


.00 




.01 




.05 




.76 


K34 


Course pace (too slow- too fast) 


.05 


.01 - 


-.13 




.09 


.09 


.05 


.39 




.09 




.43 



NOTE: rtie oblique factor analysis was performed with the commercially available SPSS routine 
(Nie^ et. al. ^ 1975). Factor loadings in the boxes are the loadings for Items designed to 



Table 3 
Factor Analysis of the Endeavor 

Evaluation Items (paraphrased) 
I PRESENTATION CLARITY 
Fl - Presentations clarified materials 
F2 - Presently clearly & siammarized 
. F3 — Made good use of examples 
ir WORKLOAD 

F4 - Students had to work hard 

F5 - Course required a lot of time 

f6 - Course workload was heavy 

III PERSONAL ATTENTION 

F7 - Listened & was willing to help 
F8 - Able to get personal attention 
F9 - Concerned about student difficulties 

IV CLASS DISCUSSION 

FIO- Class discussion was welcome 

Fll- Students encouraged to participate 

F12- Encouraged students to express ideas 

V ORGANIZATION/PLANNING 

F13- Presentations planned in advance 
F14~ Provided detailed course schedule 
F15- Activites orderly scheduled 

VI GRADING 

F16- Grading was fair & impartial 

F17- Grading reflected student performance 

F18- Grading indicative of accomplishments 

VII STUDENT ACCOMPLISHMENTS . 
F19- Understood advanced material 
F20- Ability to analyze issues 

F21- Increased knowledge & competence 



Items (N=316 sets of ratings) 
Factor Pattern Matrix 



I 


II 


III 


IV 


V 


VI 


Vli 


T516 


.006 


.075 


.132 


.079 


.044 


.395 


.545 


-.011 


.113 


.091 


.146 


-.018 


.344 


.452 


-.010 


.082 


.127 


.124 


.129 


.234 



.075 
.005 
.058 



,813 
.900 
833 



-.029 -,039 -.008 
.069 .027 -.004 
-.016 .051 .126 



.061 .031 
.184 .041 
.113 .025 



.709 
.591 
.620 



.115 
.159 
.193 



.042 
.088 
.050 

.520 
.047 
.181 



.025 
.015 
.044 

.053 
.033 
.068 



.245 

.003. 
.221 



.180 
.006 
.074 



.690 
.894 
.724 



-.020 
.058 
.135 

-.055 
.045 
.092 



.073 
.027 
.024 



.319 
.578 
.601 



.161 .020 .077 .050 -.075 
.075 .011 .080 .044 .163 
.046 .006 .044 .048 .117 



.223 -.278 
073 .092 



.100 .046 .222 
.039 .050 .184 



.068 -.038 
-.049 .036 
.014 .066 

.074 ;.095 
.155 -.095 
.055 .223 

.089 .075 
.042 -.039, 
.026 .099 

.039 -.092 
.006 .100 
.162 .022 



702 
,844 
730 



058 



.055 .092 .028 .141 



.082 
.160 
.079 



075 
-.004 

103 

.351 
637 
.696 



NOTE: The oblique factor analysis was perfored with the commercially available SPSS 
routine (Nie, et. al., 1975). Factor loadings in the boxes are the loadings for items 
designed to measure each factor. 



Table 4 

A Multitrait-Multiinethod Matrix of Correlations to;; SEEQ and Endeavor Factors (N=316 sets of ratings) 

SEEQ Evaluation Factors 
HI. Mil. Mill MIV MV MVI MVII MVII MIX ^ : 

(92) 

55 (95) 
52 '5 60 (93) 



SEEQ Evaluation Factors 



Learning/Value (MI) 
Instructor Enthusiasm (Mil) 
Organization/Clarity (Mill) 
Group Interaction (MIV) 
Individual Rapport (MV) 
Breadth of Coverage (M VI) 
Examinations/Grading (M VII) 
Assignments/Readings (M VIII) 
Workload/Difficulty (H IX) 

Endeavor Evaluation Factors 
Presentation Clarity (F I) 
Workload (F II) 
Personal Attention (F III) 
Class Discussion (F IV) 
Organization/Planning (P v) 
Grading (F VI) 

Student Accomplishiaents (F VII) ' 



26 . l\ 24 (94) 

31 47 33 54 

39 49' 47 42 40 (88) 

46 52 48 33 32 46 (81) 

37 15 35 22 18 33 39 (84) 

06 -04 rl5 -05 -03 -CI 20 07 (91) 

SEEQ Evaluation Factors 

MI Mil Mill MIV MV MVI MVII MVIII MIX 

47 71 [§2] 23 35 49 55 32 -13 

14 06 -02 05 02 OS 32 20 Qs] 

40 56 41 63 57 43 29 -05 
29 45 20 S 57 39 33 22 -03 
58 59 |68] 26 35 56 51 39 06 
39 31 43 28 39 36 [slj 50 -04 
8OI 63 70 33 37 49 56 39 -10 



Endeavor Evaluation Factors 

FI FII FIIIFIV FV FVI FVII 
(92) 

00 (94) 

43 04 (90) 

23 05 60 (91) 

67 16 43 21 (85) 

31 06 /40 25 41 (90) 

60 03/ 44 29 60 35 (85) 



^1 



ERIC 



00 



Table 5 



Factor Analysis of the Combined Set of Items 

Evaluation Items (paraphrased) 
I LEARNINGAAIUE 

Ml Course challenging & stimulating 
M2 Learned something valuable 
M3 Increased s.'ibject interest 
F21 Increased knowledge & competence 

ir INSTRUCTOR ENTHUSIASM 

M5 Enthusiastic about teaching 

M6 'Dynamic and energetic 

M7 'EnhancecJ presention with humor 

M8 Teaching style held your interest 

III PRESENTATION CLARITY 
M9 Lecturer explanations were clear:, 
MIO Maiierials well explained & prepared 
Fl Presentations clarified materials 
F2 Presented clearly & summarized 
F3 Made good use of examples 

IV • PLANNING/OBJECTIVES 
Mil Course objectives stated & pursued 
F14 Provided detailed course schedule 

07 Aims & objectives clear to students 

08 Announced goals &/or criteria 

V GROUP INTERACTION 
M13 Encouraged class discussions 
M14 Students shared knowledge/ideas 
M16 Encouraged expression of ideas 
Flo Class discussion was welcome 

VI INDIVIDUAL RAPPORT 

M17 Friendly towards individual students 
M18 Welcomed seeking help/advice 
M20 Accessible to individual students 
F7 Listened & was willing to help 

VII BREADTH OF COVERAGE 
M21 Contrasted various implications 
M22 Gave background of ideas/concepts 
M23 Gave different points of view 
M24 Discussed current developments 

VIII GRADING 

M26 Evaluation methods fair/appropriate 
F16 Grading was fair & impartial 
F17 Grading reflected student performance 
FIB Grading indicative of accomplishments 

.IX AS S IGNMENTS/READINGS 
M28 Readings/texts were valuable 
M29 They contributed to understanding 
02 They encouraged further exploration 
05 They were integrated into the course 

X -WORKLOAD/DIFFICULTY 
M32 Course difficulty was : (Easy-Hard) 
M33 Course workload was: (Light-Heavy) 
F4 Students had to work hard 
F5 Course required a lot of time 

M30 OVERALL COURSE RATING 
M31 OVERALL . INSTRUCTOR RATING 



( N = 316 sets or ratings ) 



I 


II 


III 


IV 


V 


VI 


VII 


VIII 


IX 


X 


Tie 


.11 


.15 


.07 


.04 


.03 


.03 


.03 


.10 


.13 


.62 


.04 


.05 


.13 


.13 


-.01 


.05 


.09 


.09 


.04 


.57 


.12 


.08 


.09 


.00 


.08 


.12 


.05 


.10 


. 


.55 


.08 


. .19 


.08 


.07 


.00 


.09 


.O.'i 


.09 


-.01 



.07 


.57 


.14 


.01 


.12 


,11 


.09 


.02 


.00 


-.01 


.06 


.72 


.06 


l07 


.03 


.10 


.11 


.06 


.03 


.04 


.11 


.36 


.13 


.09 


.13 


.19 


.10 


.03 


.03 


-.08 


.16 


.39 


.34 


.12 


' .03 


.12 


-io 


.04 


.03 


.00 



.02 


.26 


.55 


.16 


.07 


.03 


.04 


.06 


.07 


-.05 


.03 • 


.23- 


.43 


.22 


.04 


.00 


.12 


.12 


.13 


-.02 


.14- 


-.29 


.48 


.12 


.06 


.02 


.10 


.02 


.08 


-.04 


.08 

.ol. 


>20' 


.62 


.16 


.08 


.04 


.03 


-.03 


.12 


-.03 


.19 


.37 


.09 


.03 


.04 


.22 


.09 


.16 


-.03 



.21 


.08 


.24 


.36 


-.02 


.14 


.15 


.08 


.05 


.01 


.06 


.00 


.72 


.01 


-.03 


.08 


-.02 


.03 


.15 


.06 


.32 


.41 


-.03 


.08 


.07 


.12 


.05 


.07 


.00 


.17 


.58 


.05 


.11 


.04 


.11 


.04 



-.02 


.05 


-.02 


.06 


.82 


.05 


.07 


.03 


.13 


.01 


.06 


.12 


.01 


.70 


.05 


.14 


.06 


.05 


.10 


.08 


-.02 


.06 


.62 


.27 


.12 


.00 


-.07 


.08 


.05 


.03 


-.03 


.70 


.18 


.00 


.09 


.05 



.03 
*02 
.00 
.03 



.01 
.03 
.02 



.08 


.13 


.07 


.03 


.18 


.57 


.06 


.02 


-.06 


.08 


.10 


.04 


.07 


.23 


.64 


.04 


.03 


-.02 


.12 


.09 


.00 


.07 


.14 


.53 


.13 


.13 


.13 


.06 


.15 


.06 


.03 


.14 


.58 


.10 


.10 


-.04 



.07 



.01 


.12 


.08 


.14 


.04 


.08 


.04 


.09 


.03 


.03 


.05 


.02 


.16 


.10 


..05 


.10 


.01 


-.02 


.14 


-.17 


.02 


.06 


.13 


-.02 


.05 


.11 


-.09 


.21 


.08 


.00 


-.01 


.09 


.03 


-.06 


-.03 


.12 


.10 


.02 


-.03 


-.09 


.27 


.29 


-.14 


.20 


.00 


-.13 


.14 


.19 


.07 


-.11 


.03 


-.03 


.12 


-.10 


-.02 


.11 
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KOTE: The oblique factor analysis was performed with the commercially available SPSS routine 
(Nie, et. al. , 1975). Factor loadings in the boxes are the loadings for items designed to 
measure each factor. 
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